This would be a typical design of a \(ConvNet \). All of the work and designing a convolutional neural net is selecting its \(hyperparameters \): deciding what's the filter size, what's the stride, what's the padding and how many filters to use. We will give some suggestions later and some guidelines for how to make these choices. At the moment, one thing to take away from this is that as we go deeper in the neural network, typically we start off with larger images \(39\times 39 \), and then the height and width will stay the same for a while and gradually trend down as we go deeper in the neural network. That is, the size has gone from \(39 \) to \(37 \) to \(17 \) to \(7 \), whereas the number of channels generally increases (from \(3 \) to \(10 \) to \(20 \) to \(40 \)).

In this post we will make a Neural Network overview. We will see what is the simplest representation of a Neural Network and how deep representation of a Neural Network looks like. You may have heard that the perceptron is the simplest version of a Neural Network. The perceptron is a one layer Neural Network with the \(step\) activation function . In the previous posts we have defined a Logistic Regression as a single unit that uses \(sigmoid\) activation function.

We're going to take this \(400 \) units and build the next layer with \(120 \) units. So, this is actually our first \(Fully \enspace connected \) layer. In this layer we have \(400 \) units densely connected to \(120 \) units. This \(Fully \enspace connected \) layer is like the single neural network layer. Hence, this is just a standard neural network where you have a weight matrix that's called \(W \) of a dimension \(120 \times 400 \).

Why do we need a random initialization? In other words unit1 and unit2 are symmetric, and it can be shown by induction that these two units are computing the same function after every iteration of training. Even if we have a lot of hidden units in the hidden layer they all are symetric if we initialize corresponding parameters to zeros. To solve this problem we need to initialize randomly rather then with zeros. And then we can initialize \(b_1\) with zeros, because initialization of \(W_1\) breaks the symmetry, and unit1 and unit2 will not output the same value even if we initialize \(b_1\) to zero.

Therefore, through learning the Deep Learning tools, we will be able to invent new products and applications. Even if we are not concerned with the development of the Computer Vision systems, creative neural network architectures from this research field could inspire us to create many other methods in the fields of speech recognition, text processing or audio processing.

In this post we will see what are the building blocks of a Deep Neural Network. We will pick one layer, for example layer \(l \) of a deep neural network and we will focus on computatons for that layer. Calculation of the forward pass for layer \( l \) we get as we input activations from the previous layer and as the output we get activations of the current layer, layer \(l \). It is good to cache the value of \( z {[l]} \) for calculations in backwardpass. Backward pass is done as we input \(da {[l]} \) and we get the output \(da {[l-1]} \), as presented in the following graph.